Topics in Tweets: A User Study of Topic Coherence Metrics for Twitter Data

摘要

Twitter offers scholars new ways to understand the dynamics\udof public opinion and social discussions. However, in order to understand\udsuch discussions, it is necessary to identify coherent topics that have been\uddiscussed in the tweets. To assess the coherence of topics, several auto-\udmatic topic coherence metrics have been designed for classical document\udcorpora. However, it is unclear how suitable these metrics are for topic\udmodels generated from Twitter datasets. In this paper, we use crowd-\udsourcing to obtain pairwise user preferences of topical coherences and to\uddetermine how closely each of the metrics align with human preferences.\udMoreover, we propose two new automatic coherence metrics that use\udTwitter as a separate background dataset to measure the coherence of\udtopics. We show that our proposed Pointwise Mutual Information-based\udmetric provides the highest levels of agreement with human preferences\udof topic coherence over two Twitter datasets.